Priority system and method for processing standardized tests

ABSTRACT

Systems and methods are provided for prioritizing the processing of standardized tests. One aspect of the present subject matter relates to a method for processing completed standardized tests. In one embodiment, test-processing priority information is received, and is associated with machine-readable identifiers for standardized tests. Completed standardized tests are received, and are identified by reading machine-readable identifiers. The identified standardized tests are processed according to the associated test-processing priority information such that higher priority tests are automatically processed before lower priority tests. Other aspects are provided herein.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of application Ser. No. 10/894,560,filed Jul. 29, 2004, which is a continuation of application Ser. No.10/153,444, filed on May 21, 2002, now issued as U.S. Pat. No.6,772,081, the specifications of which are incorporated herein byreference in their entirety.

This application is related to the following commonly assigned U.S.patent applications which are herein incorporated by reference in theirentirety: “System For Programmable Presentment Of Scoring Tasks To APlurality Of Reviewers,” Ser. No. 10/140,769, filed on May 7, 2002; and“Integrated System For Electronic Tracking And Control of Documents,”Ser. No. 10/140,768, filed on May 7, 2002.

TECHNICAL FIELD

This application relates generally to systems and methods for electronictracking and control of documents, and more particularly to electronictracking and control of standardized tests within a standardized testmanagement system.

BACKGROUND

One method for evaluating the knowledge or skill of a person includesthe use of standardized tests. For example, standardized tests are usedto monitor the academic progress of students. Some standardized testsincorporate objective test questions that are answered by filling in anappropriate oval in an associated bubble-type answer sheet using apencil. Some standardized tests include open-ended questions. Examplesof open-ended questions in these tests include essay questions,fill-in-the-blank questions, fill-in-the-map questions, math questions,and motor skill questions in which a test taker draws geometric shapes.Open-ended questions, also referred to herein as scoring tasks or testitems, are evaluated and scored by a human reviewer, also referred toherein as a reader.

A large number of completed standardized tests are received at atest-processing center from a number of test-taking centers. Manytesting projects have various time constraints. These time constraintsprovide a complicating factor in processing the large number of tests.

There is a need in the art to provide improved systems and methods forprocessing completed standardized tests.

SUMMARY

The above mentioned problems are addressed by the present subject matterand will be understood by reading and studying the followingspecification. The present subject matter relates to systems and methodsfor prioritizing the processing of standardized tests. The presentsubject matter allows a test process to follow external timeconstraints, such as those time constraints provided by the customer.The present subject matter also allows a test process to follow internaltime constraints, such as those time constraints designed forimplementing efficient processing of the standardized tests. In oneembodiment, external and internal refer to those time constraintsprovided outside of or within, respectively, a test-processing center.

One aspect of the present subject matter relates to a method forprocessing completed standardized tests. In one embodiment,test-processing priority information is received, and is associated withmachine-readable identifiers for standardized tests. Completedstandardized tests are received, and are identified by reading themachine-readable identifiers on the tests. The identified standardizedtests are processed according to the associated test-processing priorityinformation, such that higher priority tests are automatically processedbefore lower priority tests. In one embodiment, boxes of test bookletsare received from a plurality of test-taking centers, and criteria arereceived to prioritize the processing of the test booklets. Bar codesfor the received boxes are read to prioritize the processing of thereceived boxes according to the received criteria. Bar codes for thetest booklets contained within the processed boxes are read toprioritize processing of the test booklets according to the receivedcriteria. The test booklets are processed in preparation for scoringtest items in the test booklets according to the received criteria, suchthat higher priority test booklets are processed before lower prioritytest booklets. The test items are scored according to the receivedcriteria.

According to one embodiment, completed test items are electronicallyreceived from test-taking centers, and criteria are received toprioritize the processing of the test items. The test items areprocessed in preparation for scoring, and the test items are scoredaccording to the criteria.

One aspect of the present subject matter relates to a system forprocessing completed standardized tests. According to one embodiment,the system includes reading device(s) and a server connected to thereading device(s). The reading device(s) machine-read test identifiersfor tests. The server includes a database module and at least oneapplications module. The database module associates test identifierswith test-processing priority information. The applications module(s)processes standardized tests according to the test-processing priorityinformation, such that higher priority tests are processed before lowerpriority tests.

These and other aspects, embodiments, advantages, and features willbecome apparent from the following description of the present subjectmatter and the referenced drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system for processing standardized tests thataccounts for priority, according to one embodiment of the presentsubject matter.

FIG. 2 illustrates one embodiment of a method for processingstandardized tests that accounts for priority.

FIG. 3 illustrates one embodiment of a method for processing boxes ofstandardized tests that accounts for priority.

FIG. 4 illustrates one embodiment of a method for processing testbooklets that accounts for priority.

FIG. 5 illustrates one embodiment of a method for processing test itemsthat accounts for priority.

FIG. 6 illustrates one embodiment of a method for processing boxes ofstandardized test booklets that accounts for priority.

FIG. 7 illustrates a process flow for one embodiment of a system forprocessing standardized tests.

FIG. 8 illustrates one embodiment of a method for processing documentsshown in the process flow of FIG. 7.

FIG. 9 illustrates one embodiment of a system for processingstandardized tests.

FIG. 10 illustrates one embodiment for receiving test documents fromtest-taking centers.

FIG. 11 illustrates one embodiment for processing box(es) as shownwithin the method of FIG. 9.

FIG. 12 illustrates one embodiment for processing test documents asshown within the method of FIG. 10.

FIG. 13 illustrates one embodiment for processing test documents asshown within the method of FIG. 10.

FIG. 14 illustrates one embodiment for processing test documents asshown within the method of FIG. 10.

FIG. 15 illustrates one embodiment of a standardized test managementsystem.

FIG. 16 illustrates one embodiment of a method for assigning a pluralityof electronic test items to a plurality of readers.

FIG. 17 illustrates one embodiment for the reader requirements of FIG.16.

FIG. 18 illustrates one embodiment for the reader rules of FIG. 16.

FIG. 19 illustrates one embodiment for the priority rules of FIG. 16.

FIG. 20 illustrates one embodiment of a system for distributing aplurality of test items to a plurality of readers.

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawingswhich show, by way of illustration, specific aspects and embodiments inwhich the present subject matter may be practiced. In the drawings, likenumerals describe substantially similar components throughout theseveral views. These embodiments are described in sufficient detail toenable those skilled in the art to practice the present subject matter.Other embodiments may be utilized and structural, logical, andelectrical changes may be made without departing from the scope of thepresent subject matter. The following detailed description is,therefore, not to be taken in a limiting sense, and the scope of thepresent subject matter is defined only by the appended claims, alongwith the full scope of equivalents to which such claims are entitled.

The present subject matter provides systems and methods for prioritizingthe processing of standardized tests. The present subject matter allowsa test process to accommodate time constraints, such as external timeconstraints and/or internal time constraints.

FIG. 1 illustrates a system for processing standardized tests thataccounts for priority, according to one embodiment of the presentsubject matter. In the illustrated embodiment, as test-taker (e.g., astudent) 102 completes a standardized test 104 at a test-taking center(e.g., a school) 106. In this embodiment, the standardized tests includetest booklets. Each test booklet includes a number of pages that containa number of test items or questions.

Completed test booklets are boxed, illustrated at 108, for shipping to atest-processing center 110. The boxes include an identifier 112, such asa bar code for example. Upon arriving at the test-processing center 110,the boxes of test booklets are unloaded at 114. The test booklets areremoved from the boxes and sorted at 116. At 120, the test booklets arecut into loose pages. These loose pages are reconciled to ensure thatall of the pages for each test booklet are accounted for. Readingdevices 122, 124 and 126, such as bar code scanners for example, areused to read the identifiers 123 and identify the boxes, read theidentifiers 125 and identify the test booklets, and read the identifiers127 and identify the pages. In one embodiment, the optical markrecognition (OMR) scanner identifies the identifying markings for thepages.

The test pages are graded or scored at 128. In one embodiment, objectivescoring tasks, such as multiple choice questions for example, are scoredusing optical mark recognition (OMR) scanners 130. In one embodiment,open-ended scoring tasks are scanned at scanning stations 132, arestored in a queue, and are distributed by dealer 134 to human readers135 who evaluate the open-ended scoring tasks. Reports 136 of the scoreresults are provided at 137.

A server in the test-processing center includes priority information. Asillustrated via lines 138, 140, 142, 144 and 146, the priorityinformation is available at various places along the process. In oneembodiment, for example, the reading device(s) 122 determine which ofthe boxes should proceed for further processing before other boxes. Inone embodiment, the reading device(s) 124 determine which of the testbooklets should proceed for further processing before other testbooklets. In one embodiment, the reading device(s) 126 determine whichof the pages (or test items on the pages) should proceed for furtherprocessing before other pages (or test items on the pages). In oneembodiment, for example, the priority information is used in the scoringsystem 128 to determine which test items should be scored before othertest items. In one embodiment, for example, the priority information isused to determine which reports should be provided before other reports136.

FIG. 2 illustrates one embodiment of a method for processingstandardized tests that accounts for priority. The illustration shows afirst location 202, represented as a test-processing center, where thestandardized tests are received, processed and reported out. Theillustration also shows a second location 204, represented as atest-taking center, where the standardized tests are completed by testtakers and sent to the test-processing center. According to variousembodiments, the test-taking center 204 includes a plurality oftest-taking centers 202 at a plurality of locations. Additionally,according to various embodiments, the test-processing center includes aplurality of test-processing centers at a plurality of locations.

Completed tests, illustrated at 206, are received at the test-processingcenter at 208, and are identified at 210. In one embodiment, the testsare distributed, shown using line 212, using a courier service. In oneembodiment, the tests are distributed in boxes from the test-takingcenter. In one embodiment, the tests include test booklets that containa number of test items or questions, and these test booklets aredistributed in boxes from the test-taking center. In one embodiment, thetests are distributed using transmission signals, such as electronicdigital signals within the Internet computer network.

The received tests are identified using machine-readable identifiers.Machine-readable identifiers are identifying labels capable of beingread by computers or other devices. In one embodiment, themachine-readable identifiers include bar codes. In one embodiment, themachine-readable identifiers include electronic identifiers for atransmitted signal representing the completed tests. One example of atransmitted signal includes a digital transmission. For example, in oneembodiment, the electronic identifiers include tags that are capable ofbeing read by a computer on the Internet global network. Systems andmethods that use machine-readable identifiers that are based on othertechnology also are within the scope of the present subject matter.

At 214, test-processing priority information is received at thetest-processing center. In one embodiment, this test-processing priorityinformation indicates which tests should be processed first throughvarious processing stages. According to various embodiments, thistest-processing priority information indicates which boxes of testdocuments are a priority, which test documents are a priority, and/orwhich test items are priority in various test-processing stages.

In one embodiment, customer-provided priority information, representedat 216, is received. Customer-provided priority information 216 includesthe customer's requirements for a test project. In one embodiment, forexample, a state agency is able to prioritize the processing for adistrict, a school, a class, a student, a grade, a subject, a test item,a form and/or a material type.

According to various embodiments, the test-processing center providespriority information at 218 and/or 220. At 218, for example, thetest-processing center is able to apply its own internal rules forprioritizing the processing the tests. For example, some open-endedquestions are read by two different readers. After the first reader hasread the test item, one internal rule provides that the second readingof the test item has an increased priority. At 220, for example, thetest-processing center monitors the test-processing stages, and providesfeedback to appropriately prioritize the processing of tests upstream toimprove the overall efficiency of the system. For example, readers arequalified to read certain types of test items. It is desirable tomaintain a steady workflow to the readers. Should the work for aparticular reader run low, the feedback provides an indication to theupstream processes (such as box receipt and book receipt, described inmore detail below) to prioritize the processing of certain boxes, testbooklets, and test items to increase the workflow to that particularreader.

The priority information received at 214 is associated with machinereadable identifiers for tests at 222. According to various embodiments,these machine readable identifiers include identifiers for boxes oftests, test booklets, and/or test items. The association of theidentifiers and the priority information is maintained in a database.The tests that have been received at 208 and identified at 210 areautomatically prioritized at 224. Computer instructions are used toautomatically prioritize the tests. In one embodiment, the tests areautomatically prioritized based on a start command provided as auser-input. In one embodiment, the tests are automatically prioritizedat predetermined times, such as at periodic intervals (e.g. daily,hourly etc.), for example. In one embodiment, the tests areautomatically correlated as test data are received and become availableto be operated on by the computer instructions. The tests are processedat 226. In one embodiment, priority feedback information is provided at220 based on the test processing. Test results are reported at 228.

In one embodiment, the automatic prioritization of tests at 224dynamically occurs in a real time or near real time fashion. In variousembodiments, for example, a number of processing stages are used toprocess received and identified tests. The relative priorities for thesein-process tests are capable of being automatically modified at 224 whennew priority information is received at 214 and associated with theidentifiers at 222.

FIG. 3 illustrates one embodiment of a method for processing boxes ofstandardized tests that accounts for priority. In one embodiment, themethod represented by the dotted line box 330 generally corresponds tothe method represented by the dotted line box 230 in FIG. 2. Boxes oftests are received at 332. The term box, as used herein, includesvarious containers that are capable of being used to ship and receivethe test documents. The term box is used to simplify the disclosure byusing a well-known, easily-visualized shipping container. One ofordinary skill in the art will understand, upon reading andcomprehending this disclosure and the attached claims, that the presentsubject matter is not limited to boxes or to any other type of shippingcontainer, and that the term box as used in the claims is entitled to beinterpreted as encompassing a full scope of equivalent shippingcontainers.

At 334, the boxes are identified using machine-readable identifiers. Inone embodiment, the boxes are identified using bar code labels. Boxpriority information, represented generally at 336, is associated withmachine-readable identifiers for boxes at 338. At 340, the associatedpriority information is used to automatically prioritize the receivedand identified boxes for processing. The tests contained within theprioritized boxes are processed at 342. In one embodiment, priorityfeedback information, as represented at 320, is provided.

FIG. 4 illustrates one embodiment of a method for processing testbooklets that accounts for priority. In one embodiment, the methodrepresented by the dotted line box 430 generally corresponds to themethod represented by the dotted line box 230 in FIG. 2. Test bookletsare received at 444. A test booklet includes a number of test items, orquestions, assembled into a single document. Test booklets have amachine-readable document number, referred to in this context as a testbooklet identifier.

At 446, the test booklets are identified using machine-readableidentifiers. In one embodiment, the test booklets are identified usingbar code labels as test booklet identifiers. Test booklet priorityinformation, represented generally at 448, is associated withmachine-readable identifiers for test booklets at 450. At 452, theassociated priority information is used to automatically prioritize thereceived and identified test booklets for processing. The testscontained within the test booklets are processed at 454. In oneembodiment, priority feedback information, as represented at 420, isprovided.

FIG. 5 illustrates one embodiment of a method for processing test itemsthat accounts for priority. In one embodiment, the method represented bythe dotted line box 530 generally corresponds to the method representedby the dotted line box 230 in FIG. 2. Test items are received at 556. Atest item includes test questions that are to be scored. In oneembodiment, the test items are shipped from the test-taking center tothe test-processing center. In one embodiment, the test items aretransmitted via transmission signals from the test-taking center to thetest-processing center. For example, the test items are securelytransmitted through the Internet computer network in one embodiment. Thetest items have a machine-readable test item identifier. In oneembodiment, the test item identifier is a bar code located on a tangibletest item. In one embodiment, the bar code forms part of a scanned imageof the test item, such that the scanned image and bar code are able tobe transmitted via transmission signals. In one embodiment, the testitem identifier is an electronic identifier, such as a tag, for anelectronic transmission.

At 558, the test items are identified using machine-readableidentifiers, examples of which were provided above. Test item priorityinformation, represented generally at 560, is associated withmachine-readable identifiers for test items at 562. At 564, theassociated priority information is used to automatically prioritize thereceived and identified test items for processing. The test items areprocessed at 566. For example, one embodiment of processing the testitems includes scoring the test items. In one embodiment, priorityfeedback information, as represented at 520, is provided.

FIG. 6 illustrates one embodiment of a method for processing boxes ofstandardized test booklets that accounts for priority. In oneembodiment, the method represented by the dotted line box 630 generallycorresponds to the method represented by the dotted line box 230 in FIG.2. Boxes of test booklets are received at 632. At 634, the boxes areidentified using machine-readable identifiers. In one embodiment, theboxes are identified using bar code labels. Box priority information,represented generally at 636, is associated with machine-readableidentifiers for boxes at 638. At 640, the associated priorityinformation is used to automatically prioritize the received andidentified boxes for processing. In one embodiment, the physicallocation of the boxes are tracked after they have been received. Shouldthe box priority information change during processing, the higherpriority boxes are able to be found, and the test booklets in the higherpriority boxes are able to be distributed for further processing beforethe lower priority boxes.

At 646, the test booklets are identified using machine-readableidentifiers. In one embodiment, the test booklets are identified usingbar code labels as test booklet identifiers. Test booklet priorityinformation, represented generally at 648, is associated withmachine-readable identifiers for test booklets at 650. At 652, theassociated priority information is used to automatically prioritize thereceived and identified test booklets for processing. In one embodiment,the physical location of the test booklets are tracked after they havebeen received. Should the test booklet priority information changeduring processing, the higher priority test booklets are able to befound, and the test items in these higher priority test booklets areable to be distributed for further processing before the lower prioritytest booklets.

At 658, the test items are identified using machine-readableidentifiers. Examples of machine-readable identifiers for test itemswere provided above. Test item priority information, representedgenerally at 660, is associated with machine-readable identifiers fortest items at 662. At 664, the associated priority information is usedto automatically prioritize the received and identified test items forprocessing. The test items are processed at 666. For example, oneembodiment of processing the test items includes scoring the test items.In one embodiment, priority feedback information, as represented at 620,is provided. According to various embodiments, this priority feedbackinformation is used to generate the box priority information 636, thetest booklet priority information 648, and/or the test item priorityinformation 660.

FIG. 7 illustrates a process flow for one embodiment of a system forprocessing standardized tests. Test-taking centers are represented asdistrict/schools at 768. Completed standardized test materials, referredto generally herein as tests, are returned to the test-processing centerat 770. In one embodiment, the tests are returned from the test-takingcenter 768 to the test-processing center 770 in boxes, as representedgenerally by 772. In one embodiment, the tests are returned from thetest-taking center 768 to the test-processing center 770 electronicallyusing transmission signals, as represented generally by 774.

In an embodiment in which the tests are returned to the test-processingcenter in boxes 772, a box receipt process 776 is performed on theboxes. In one embodiment, the shipped boxes are identified by reading amachine-readable identifier, such as by scanning a bar code, forexample. The identified boxes that have a higher priority aredistributed for further processing before the identified boxes that havea lower priority. The figure illustrates customer-provided priorityinformation 773 being received from the test-taking center 768.

In one embodiment, the boxes contain test booklets. The test bookletsare unpacked, and a book receipt process is performed at 778. In oneembodiment, the test booklets are identified by reading amachine-readable identifier, such as by scanning a bar code or bydetecting identifying markings (e.g., filled-in ovals) on the testbooklets using optical mark recognition (OMR) logic. The identified testbooklets that have a higher priority are distributed for furtherprocessing before the identified test booklets that have a lowerpriority.

At 780, document processing is performed on the test booklets to preparethe test booklets to be scored. The test booklets include test items. Aperformance assessment process is performed at 782 to score the testitems. In one embodiment, the test items are scored using OMR logic forobjective (e.g., multiple choice) test items, and using readers foropen-ended test items. One example of a performance assessment processis provided in the co-pending, commonly assigned U.S. patent applicationentitled “System For Programmable Presentment Of Scoring Tasks To APlurality Of Reviewers,” Ser. No. 10/140,769, filed on May 7, 2002,which was previously incorporated by reference in its entirety.

In one embodiment, the priority information is provided using numericalpriority. For example, in one embodiment, a priority 2 indicates ahigher priority than a priority 4. In this embodiment, a priority 3indicates a lower priority than a priority 2, and a higher priority thana priority 4. One of ordinary skill in the art will understand, uponreading and comprehending this disclosure, that a priority level iscapable of being assigned to boxes, test booklets and/or test items.

In one embodiment, priority information is flagged at a district/schoollevel. In various embodiments, priority is capable of being set forclasses, grades, subjects and test items. According to variousembodiments, the priority information is fed to a number of systems orprocesses throughout the test-processing center. In one embodiment, arelational database system is used to relate a priority to a box oftests, a priority to a test booklet and/or a priority to a test item,such that the priority is capable of being used to efficiently processboxes, booklets and/or test items.

In an embodiment in which the tests are returned to the test-processingcenter via a transmission signal 774, document processing 780 isperformed on the transmitted tests to prepare the tests to be scored.According to various embodiments, for example, document processing fortransmitted tests includes receiving the transmission, decoding thetransmission, organizing or sorting the transmitted test items, andimage processing. The figure illustrates customer-provided priorityinformation 775 being received from the test-taking center 768. Aperformance assessment process is performed at 782 to score the testitems.

FIG. 8 illustrates one embodiment of a method for processing documentsshown at 780 in the process flow of FIG. 7. In the illustratedembodiment 880, test booklets are cut at 884 to separate individualpages. These pages include a number of test items. At 886, the pages arescanned to provide electronic images of the pages. At 888, theelectronic images are edited in preparation for scoring test itemscontained in the images. According to various embodiments, for example,the images are copied and cropped to provide individual images of eachtest item or group of test items. In other embodiments, the images arecompressed, the background of the test pages is removed, the images aresharpened and cleaned up, and/or other template images are superimposed.Other image processing is performed in other embodiments. At 890, theimages are reconciled against the test booklets to verify that all ofthe pages are in the system for each book. The reconciliation processdetermines if too few or too many pages are present. Discrepancies areinvestigated and corrected. At 892, data are validated. The validationis a higher level logic check on the data. According to variousembodiments, for example, the logic ensures that each lithocode (alsoreferred to as a security code) on test documents is unique, and thatensures that there are no duplicate precode data. According to variousembodiments, precode data include student information such as name, age,education level, class, school and other information deemed relevant forreporting purposes. In one embodiment, precode data further includeinformation regarding test requirements for individual test-takers.Examples of test requirements include, but are not limited to, the needfor Braille or large print tests for individual test-takers. The precodedata are associated with the security number. According to variousembodiments, various precode data are printed on the tests along withthe security code.

FIG. 9 illustrates one embodiment of a system for processingstandardized tests. The illustrated system generally includes atest-processing center 901, which generally includes a server 903 incommunication with one or more reading devices 905. Machine-readabletest identifiers 907 for tests are read by the reading device(s) 905. Inone embodiment, the machine-readable identifiers 907 include bar codes,and the reading device(s) 905 include bar code scanners. In oneembodiment, the machine-readable identifiers 907 include electronicidentifiers for electronically transmitted tests. As one of ordinaryskill in the art will understand upon reading and comprehending thisdisclosure, the present subject matter is not limited to a particularmachine-readable technology.

In one embodiment, the tests include a number of test items 911. Thetest items 911 include item identifiers 913. The item identifiers aremachine-readable identifiers such a bar codes or electronic identifiers,for example. In one embodiment, a number of the test items 911 form testbooklets 915. The test booklets 915 include booklet identifiers 917 thatare machine-readable identifiers such electronic identifiers, forexample. In one embodiment, the test booklets 915 are shipped in boxes919. The boxes 919 include box identifiers 921 that are machine-readableidentifier such as bar codes, for example.

The server 903 includes a database module 923 and at least oneapplications module 925. The database module 923 associates testidentifiers 927 with test-processing priority information 929. Accordingto various embodiments, the database module 923 associates test itemidentifiers 913 with priority information for test items, associatestest booklet identifiers 917 with priority information for testbooklets, and/or associates box identifiers 921 with priorityinformation for boxes. The applications module(s) 925 process the testsaccording to the test-processing priority information, such thathigher-priority tests are processed before lower-priority tests.

In one embodiment, the test-processing priority information 929 includespriority information such as test-processing feedback information,provided by the test-processing center. In one embodiment, thetest-processing priority information includes customer-provided priorityinformation. One embodiment of the server 903 includes a communicationmodule 931 for receiving test-processing priority information from acustomer 933, such as a test-taking center, for example. In oneembodiment, the communication module communicates with the customerthrough the Internet computer network.

Various embodiments of the server include a box receipt module 935and/or a book receipt module 937. The box receipt module 935 identifiesreceived boxes 919 by reading the identifiers 921 on the received boxes919 using the reading device(s) 905. The box receipt module 935 alsoprioritizes the processing of the received boxes according to thetest-processing priority information 929. The book receipt module 937identifies received test booklets 915 by reading the identifiers 917 onthe received booklets 915 using the reading device(s) 905. The bookreceipt module 937 also prioritizes the processing of the received testbooklets according to the test-processing priority information 929.

The present subject matter provides a system for prioritizing theprocessing of standardized tests. Various embodiments for processingstandardized tests include a number of sub-processes. Two sub-processembodiments are described below as examples. One sub-process, as shownand described with respect to FIGS. 11-15, relates to receiving boxes oftest booklets. Another sub-process, as shown and described with respectto FIGS. 16-20, relates to the presentation of scoring tasks to readers.

FIG. 10 illustrates one embodiment for receiving test documents fromtest-taking centers. In this embodiment 1016, one or more boxes of testdocuments (e.g., test booklets) sent from the test-taking centers arereceived at the test-processing center at 1032. The received box(es) areprocessed at 1034. In one embodiment, processing the received box(es)includes verifying that box(es) received at the test-processing centercorrespond to the boxes shipped from the test-taking centers. Accordingto various embodiments, the verification process uses a variety ofinformation sources including, among other things, documentation shippedwith the boxes, courier tracking data, and/or information providedthrough an on-line update from the test-taking centers, identifying theboxes that had been shipped. One of ordinary skill in the art willunderstand, upon reading and comprehending this disclosure, that thepriority information is capable of being used to prioritize theprocessing of the boxes and/or test documents.

FIG. 11 illustrates one embodiment for processing box(es) as shown at1034 within the method of FIG. 10. According to this embodiment 1134,shipping data is received at 1138. According to various embodiments,this shipping data is taken from, for example, documentation shippedwith the boxes, courier tracking data, and/or information providedthrough an on-line update from the test-taking centers identifying theboxes that had been shipped. A database is updated with the receivedshipping data at 1140.

The received boxes are identified at 1142, and are validated against thereceived shipping data at 1144. One validation method involves countingthe shipped boxes and the received boxes, and verifying that the numberof shipped and received boxes are the same. One validation methodinvolves labeling each of the shipped boxes with a unique identifier,and using the unique identifier to verify that a shipped box wasreceived. According to one embodiment, the boxes are labeled with barcodes that represent a unique box identifier.

The database is updated with the validation results at 1146. Thisinformation is used to indicate discrepancies, that is, to indicatewhether the one or more shipped boxes sent from the test-taking centerhave been received at the test-processing center. At 1148, the shipmentstatus of the shipped and received boxes is reported to the appropriatepeople for detecting and resolving discrepancies.

At 1150, the boxes are assigned to Work-In-Progress (WIP) groups forfurther processing. According to one embodiment, the boxes of testdocuments are assigned to labeled pallets, which form the WIP groups. Inone embodiment, the pallets are labeled with bar codes that representunique pallet identifiers. Should the priority information change, theappropriate pallet or box on a pallet is capable of being found andprocessed expeditiously.

FIG. 12 illustrates one embodiment for processing test documents asshown at 1036 within the method of FIG. 10. In this embodiment 1236, asecurity data record is identified for each of the test documents beingprocessed at 1252, and a precode data record, if available, isidentified for each of the test documents being processed at 1254. Invarious embodiments, the security data record and/or the precode datarecord is/are identified by reading a bar code.

A precode data record may not always be present on a received testdocument, because test-takers can move, for example, or can otherwise beadded or deleted. In one embodiment, if a test-taker does not receive atest document with precode data, the test-taker completes one of anumber of “extra” test documents that are distributed to the test-takingcenter along with the precoded test documents. Each of these extra testdocuments includes a unique secure test document identifier. A bar codeor other marking indicates that a completed test document is notprecoded, and thus needs to be processed to determine the studentinformation that is to be associated with the secure document. In oneembodiment, a test administrator from the test-taking center updates thetest-taker population data with information identifying whether atest-taker completed a test document, and providing information forthose test-takers who were not accounted for in the precode data.

At 1256, one or more scanned images for each of the received testdocuments are created. The scanned image(s) of the test documents areprocessed at 1258. One of ordinary skill in the art will understand,upon reading and comprehending this disclosure, that according tovarious embodiments, relative priorities affect the order of creatingand processing scanned images.

In one embodiment, the scanned image(s) are processed using variousimage processing software and/or filters to score objective scoringtasks. In one embodiment, the scanned image(s) are processed usingvarious image processing software and/or filters to score open-endedscoring tasks. In one embodiment, the scanned image(s) are processedusing various image processing software and/or filters to score bothobjective and open-ended scoring tasks. In one embodiment, theopen-ended scoring tasks are processed using artificial intelligence ina computer-based process. For example, handwritten answers are capableof being detected and deciphered into an answer, and artificialintelligence is applied to the answer to score the test item.

FIG. 13 illustrates one embodiment for processing test documents asshown at 1036 within the method of FIG. 10. In this embodiment 1336, asecurity data record is identified for each of the test documents beingprocessed at 1352, and a precode data record, if available, isidentified for each of the test documents being processed at 1354. Invarious embodiments, the security data record and/or the precode datarecord is/are identified by scanning a bar code.

In the illustrated embodiment, an optical mark recognition (OMR) processis used to process objective scoring tasks at 1360. An OMR process iscapable of detecting ovals or circles that have been filled in using acarbon-based pencil, for example. In one embodiment, the OMR processuses a filter to identify the position of the marks on the testdocument.

In the illustrated embodiment, scanned images of open-ended scoringtasks are created at 1362. Examples of open-ended questions in thesetests include essay questions, fill-in-the-blank questions,fill-in-the-map questions, math questions, and motor skill questions inwhich a test taker draws geometric shapes. The open-ended scoring tasksare distributed to human readers at 1364. In one embodiment, theopen-ended scoring tasks are distributed to human readers using a dealersystem provided in the patent application entitled “System ForProgrammable Presentment Of Scoring Tasks To A Plurality Of Reviewers,”Ser. No. 10/140,769, filed on May 7, 2002, which has been previouslyincorporated by reference in its entirety. One aspect of one embodimentof the dealer system involves, among other things, distributing theopen-ended scoring tasks to readers based on a priority.

FIG. 14 illustrates one embodiment for processing test documents asshown at 1036 within the method of FIG. 10. In this embodiment 1436,received test documents are sorted according to project requirements at1466, and the sorted test documents are processed at 1468. In oneembodiment, the received test documents are sorted into boxes (orreboxed) with like or similar test documents. In one embodiment,processing sorted test documents includes identifying a security datarecord and a precode data record as illustrated at 1470, and at 1472,verifying that the test documents were sorted according to the projectrequirements.

In one embodiment, as illustrated at 1474, processing the sortedplurality of standardized test documents includes providing a report forshowing a status of the plurality of test documents contained in thereceived box(es). According to one embodiment, the status reportsinclude verification information as to whether the test documents weresorted to project requirements. In one embodiment, as illustrated at1476, processing the sorted plurality of standardized test documentsincludes providing problem resolution screens. These problem resolutionscreens, for example, guide a person through various decisions and/oractions and provide the user with options for resolving thediscrepancies. In one embodiment, the problem resolution screens providesome of the more likely solution possibilities first, before providingother solution possibilities. Examples of discrepancies includedocuments that were not sorted to project requirements as represented at1472, received documents that do not match the distributed documents, ordistributed documents that do not match the received documents. In oneembodiment, as illustrated at 1478, processing the sorted plurality ofstandardized test documents includes providing user-defined queries. Oneembodiment includes queries for tracking the secure test documentsthroughout processing.

FIG. 15 illustrates one embodiment of a standardized test managementsystem. The figure illustrates a test-processing center 1510 wherestandardized tests are assembled, distributed to test-taking centers,collected from test-taking centers and processed. FIG. 15 furtherillustrates a test-taking center 1512 where standardized tests arecompleted by test-takers. One of ordinary skill in the art willunderstand, upon reading and comprehending this disclosure, that theillustrated test-processing center encompasses one or more physicaltest-processing centers, and that the illustrated test-taking centerencompasses one or more physical test-taking centers.

According to one embodiment, the tests are distributed and received viaparcel post or other courier service. The test documents are shipped inboxes 1538 or other shipping containers. A number of the distributedtest documents 1540 are identified with security data 1544 and precodedata 1546. The received test documents 1542 may or may not be shipped inthe same boxes in which they were distributed. In one embodiment, theboxes are labeled with a unique identifiers, such as a bar code. Inaddition to the security data 1544 and the precode data 1546, thereceived test documents 1542 include completed objectivequestions/activities 1548 and/or completed open-endedquestions/activities 1550.

In one embodiment, the tests are distributed and received via acommunication channel, such as through the Internet global network. Inone embodiment, test-takers 1536 complete the test on-line through acommunication channel. One of ordinary skill in the art will understand,upon reading and comprehending this disclosure, that in variousembodiments the test-processing center 1510 includes one or morephysical sites, and the test-taking centers 1512 include one or morephysical sites.

The illustrated test-processing center 1510 includes a server 1514 thatincludes a processor 1516 in communication with a memory 1518. Theillustrated server 1514 includes various functional modules. In oneembodiment, these modules are contained within the memory 1518 andinclude instructions to be operated on by the processor 1516. One ofordinary skill in the art will understand that these functional modulescan be formed from software, hardware, or various combinations ofsoftware and hardware.

In one embodiment, the server 1514 includes a database module 1520 forassociating secure document data with test-taker data. One embodiment ofthe server includes an on-line enrollment module 1522 for a testadministrator to populate test-taker data through on-line communication.One embodiment of the server includes a verification module 1524 forverifying received secure documents against distributed secure documentsand the populated test-taker data. The verification module 1524 iscapable of performing other verifications to promote the integrity ofthe system. One embodiment of the server includes a reporting module1526 for reporting verification results from the verification module.

One embodiment of the server includes a test evaluation module 1528 forprocessing test answers and collecting test results. One embodiment ofthe test evaluation module 1528 includes a dealer module 1530, such asthat provided in the patent application entitled “System ForProgrammable Presentment Of Scoring Tasks To A Plurality Of Reviewers,”Ser. No. 10/140,769, filed on May 7, 2002, which has been previouslyincorporated by reference in its entirety. The dealer module 1530distributes open-ended test items to readers 1552 and receives testscores from readers 1552. One embodiment of the test evaluation module1528 includes optical mark recognition (OMR) logic 1532 for processingobjective responses. In one embodiment, the reporting module 1526reports test results from the test evaluation module.

One embodiment of the server includes a tracking module 1534 fortracking a location of physical secure documents. According to oneembodiment, the tracking module 1534 also tracks electronic filesrepresenting secure documents.

One embodiment of the server 1514 includes a communication module 1537to provide on-line communication with one or more test-taking centers1512. One embodiment of the communication module 1537 provides on-lineenrollment communication between the test-processing center 1510 and theone or more test-taking centers 1512. One embodiment of thecommunication module 1537 provides preceding communication between thetest-processing center 1510 and the one or more test-taking centers1512. One embodiment of the communication module 1537 provides reportcommunication from the test-processing center 1510 to the one or moretest-taking centers 1512. One embodiment of the communication module1537 provides communication for updating a test-taker database from theone or more test-taking centers 1512.

One embodiment of the test-processing center 1510 includes a testassembly area 1554 or areas in which the secure test documents areassembled or otherwise prepared. The test assembly area is incommunication with the server 1514, and in particular with the database1520 in the server 1514, such that the security data and the associatedprecode data are properly printed on the secured test documents andidentified in the database 1520.

One embodiment of the test-processing center 1510 includes a testdistributing area or areas 1556, in which the tests are distributed fromthe test-processing center 1510 to the test-taking center(s) 1512. Inone embodiment, the test distributing area 1556 includes one or morehand-held scanners or scanner stations for reading bar codes thatrepresent the security data and the precode data. The test distributingarea 1556 is in communication with the server 1514, and in particularwith the database module 1520 and the tracking module 1534, to identifythe test documents that are being shipped to the test-taking center(s).According to one embodiment, the verification module 1524 verifies thatthe desired test documents are being distributed to the test-takingcenter(s). According to one embodiment, the reporting module 1526reports information regarding the shipped test documents.

One embodiment of the test-processing center 1510 includes a testreceiving area or areas 1558, in which the tests are received from thetest-taking center(s) 1512. In one embodiment, the test receiving area1558 includes one or more hand-held scanners or scanner stations forreading bar codes that represent the security data and the precode data.In one embodiment, the scanners acquire a scanned image of the testdocument for processing in the test-processing area 1560. The testreceiving area 1558 is in communication with the server 1514, and inparticular with the database module 1520 and the tracking module 1534,to identify the test documents that are being shipped from thetest-taking center(s) 1512 to the test-processing center 1510. Accordingto one embodiment, the verification module 1524 verifies that thereceived test documents correspond to the distributed test documents.According to one embodiment, the reporting module 1526 reportsinformation regarding the shipped test documents. In one embodiment, thetest-taker population is updated on-line through the communicationmodule 1537 so as to identify the test-takers 1536 who actuallycompleted the test documents.

One embodiment of the test-processing center 1510 includes atest-processing area 1560 or areas in which the completed test documentsare evaluated. In one embodiment, the test-processing area 1560 includesoptical mark recognition (OMR) equipment for evaluating objectivequestions 1548. In one embodiment, the test-processing area 1560includes scanners for scanning open-ended portions of the test documentsto be dealt to a reader 1552 and evaluated. In one embodiment, thetest-processing area 1560 includes scanners that acquire images of bothobjective 1548 and open-ended 1550 portions of the tests. Imageprocessing software processes the scanned images to evaluate theobjective and open-ended responses.

One embodiment of the test-processing center 1510 includes an inventoryarea or areas 1562. The inventory area 1510 includes a Work-In-Progress(WIP) area 1564 in which test documents are grouped and stored duringprocessing. In one embodiment, the test documents are grouped in the WIParea 1564 on labeled pallets. In one embodiment, the test documents aregrouped in the WIP area 1564 according to a project-determined priority.The WIP area 1564 is in communication with the server 1514, andparticularly with the tracking module 1534 and the database 1520 totrack the location and status within the inventory area 1562. Theverification module 1524 is capable of verifying the progress of thetest documents at various points along the process.

The inventory area 1510 also includes an archive area 1566. In oneembodiment, the archive area 1566, or a portion thereof, is locatedwithin the test-processing center 1510. In one embodiment, the archivearea 1566, or a portion thereof, is located off-site from thetest-processing center 1510. The archive area 1566 is in communicationwith the server 1514, and particularly with the tracking module 1534 andthe database 1520 to track the location and status within the inventoryarea 1562. According to one embodiment, the archive area 1566 storesphysical documents. According to one embodiment, the archive area 1566stores electronic files of test images, test results and/or reports, andthese are available for on-line retrieval. In one embodiment, the testresults are retrieved by a number of persons within the test-processingcenter 1510 using an intranet network, for example. In one embodiment,the test results are retrieved by a customer, such as testadministrators or teachers, for example.

As provided above, sub-processes for processing standardized tests arecapable of receiving and using priority information to efficientlyprocess tests. One sub-process relating to receiving boxes of testbooklets was shown and described above with respect to FIGS. 11-15.Another sub-process, as shown and described below with respect to FIGS.16-20, relates to the presentation of scoring tasks to readers.

FIG. 16 illustrates one embodiment of a method 1640 for assigning aplurality of electronic test items to a plurality of readers. Asrepresented by 1642, a number of scoring tasks are available to beassigned. In various embodiments, these scoring tasks include variousclipped portions or instances of test images. Scoring rules, representedgenerally as 1644, are applied to the scoring tasks, and high-priorityscoring tasks are assigned to qualified readers at 1646. According toone embodiment, the scoring rules 1644 include priority rules 1647 usedto determine whether the evaluation of a scoring task is a high-priorityor a lower priority. In one embodiment, the scoring rules 1644 includerules for qualifying potential readers for each scoring task asrepresented at 1648. In one embodiment, for example, qualifyingpotential readers 1648 includes applying one or more reader requirementsfor reading the scoring tasks as represented at 1650, and applying oneor more reader rules for reviewing the scoring tasks as represented at1652.

Reader requirements are described below with respect to FIG. 17.Examples of a reader requirements include requiring that a reader haveappropriate credentials to evaluate scoring tasks for a certain gradelevel such as a third grade reader, and to evaluate certain scoringtasks in certain subject matters such as math or science. Reader rulesare described below with respect to FIG. 18. Examples of reader rulesinclude a maximum percentage of a single test that can be evaluated by asingle reader, and other rules for distributing scoring tasks amongreaders according to their current workload and their capacity toevaluate additional scoring tasks.

FIG. 17 illustrates one embodiment for the reader requirements of FIG.16. In one embodiment, the reader requirements 1750 include a readerqualification rule 1754. Examples of a reader qualification rule 1754include a third grade math qualification for evaluating third grade mathscoring tasks, and an eighth grade science qualification for evaluatingeighth grade science scoring tasks, and the like. In one embodiment, thereader requirements 1750 include a team assignment rule 1756. Forexample, the scoring tasks are distributed among the members of ascoring team or scoring teams. The team members are considered to bequalified to evaluate the scoring task type, and are dealt the scoringtask if other applicable rules are satisfied. In one embodiment, thereader requirements 1750 include the number of scoring task items 1758for which each reader is qualified. This information is capable of beingused in a variety of distribution algorithms to promote efficientevaluation and processing through the distribution of an appropriatequantity and type of scoring tasks to the readers.

FIG. 18 illustrates one embodiment for the reader rules of FIG. 16. Inone embodiment, the reader rules 1852 include sorting criteria forscoring task items 1860. Examples of sorting criteria include sorting byschool and sorting by class. This information is capable of being usedin a variety of rules, such as ensuring that a single reader does notevaluate more than a certain percentage of scoring task items for aclass, for example.

In one embodiment, the reader rules 1852 include the number of itemsleft to be assigned to readers 1862, and the number of items alreadyassigned to readers 1864. This information is capable of being used toappropriately distribute the remaining test items to the qualifiedreaders. According to one embodiment, the items are distributed toprovide an equal distribution among the qualified readers. In oneembodiment, the items are distributed so as to promote efficientevaluating and processing by distributing more items to more efficientreaders. In one embodiment, the efficiency and performance of thereaders is monitored and dynamically updated based on the monitoredefficiency and performance to accurately reflect a current status and tofurther improve the efficiency of the distribution process through anautomated process. According to various embodiments, the efficiency andperformance of the readers is monitored and/or dynamically updatedperiodically or in a real time or near real-time manner.

In one embodiment, the reader rules 1852 include collaboration rules1866 to maintain the quality of the scoring process. Examples ofcollaboration rules 1866 include read-behind rules. A 10% read-behindrule requires that 10% of all scoring task items are read by a secondreader. A 100% read-behind rule requires that all of the scoring taskitems are read by a second reader. Another example of a collaborationrule 1866 requires that the second evaluation of a scoring task item isperformed by a second reader. Another collaboration rule exampleprovides a mechanism for resolving different scores from the first andsecond reader. For example, one embodiment of the mechanism requires athird reader or supervisor to evaluate scoring task items that havedifferent scores. In one embodiment, the read-behind for individualreaders is dynamic, such that more scoring task items are read a secondtime for individual readers whose scores have differed significantlyfrom other readers.

FIG. 19 illustrates one embodiment for the priority rules of FIG. 16. Inone embodiment, the priority rules 1946 include assigning a highpriority to the evaluation of predetermined “fast track items” 1968. Forexample, a school district or state may require reports of score resultsfor certain subjects or classes sooner than other reports. Thesesubjects or classes are classified as fast track items. One of ordinaryskill in the art will understand, upon reading and comprehending thisdisclosure, that a scoring task item is capable of being classified aseither a fast track item or not a fast track item, and that a scoringtask item is capable of being classified into one of many potential fasttrack levels to appropriately prioritize the evaluation of these scoringtask items. In one embodiment, the priority rules 1946 include assigninga high priority to those scoring task items that have been selected asread-behind items 1970.

FIG. 20 illustrates one embodiment of a system for distributing aplurality of test items to a plurality of readers. In one embodiment,the system 2000 operates in the test-processing center 204 shown in FIG.2. The system 2000 includes a database server 2002, illustrated as ahand score (HS) database server for processing open-ended scoring tasks,and an application server 2004 in communication with the database server2002. The figure also illustrates that the system 2000 includes a memorystorage 2006 in communication with the application server 2004 and thedatabase server 2002. One of ordinary skill in the art will understand,upon reading and comprehending this disclosure, that the memory storage2006 is capable of being organized in a number of ways. For example, thememory storage 2006 is capable of being organized as a central memorystorage for the databases and for the various applications operating onthe application server. The memory storage 2006 also is capable of beingorganized as a number of individual memory storage areas associated withthe various applications and database.

In one embodiment, the system 2000 includes a web server 2008 incommunication with the database server 2002, the application server 2004and the memory storage 2006. One example of a web server 2008 is theMicrosoft Windows NT Internet Information Server (IIS). One of ordinaryskill in the art will understand, upon reading and comprehending thisdisclosure, that the web server 2008 is capable of functioning as aninterface for both readers and supervisors, and that the web server 2008allows reader and supervisor workstations 2010 to be positioned in localand/or remote locations.

One of ordinary skill in the art also will understand, upon reading andcomprehending this disclosure, that the web server 2008 is capable offunctioning as an interface to the test-taking centers where thestandardized tests are administered. The test, or at least portionsthereof, are capable of being completed on a computer or otherelectronic device, either on-line or off-line with respect to the webserver. The answers for the test are capable of being received from thetest-taking center(s) 2012 to the web server. The scored results of thetests are capable of being reported to the test-taking center(s) 2012through the web server.

In one embodiment, the database server 2002 and the associated memorystorage, as represented by the database data 2014, include a reader file2016, and image file 2018, and a test taker file 2020. The reader file2016 identifies a plurality of readers and credentials for the readers.The image file 2018 includes images of a plurality of test items andassociates each image with one or more reader requirements and anevaluation priority. The test-taker file 2020 associates a test-takerwith one or more images of the plurality of test items.

In one embodiment, the web server 2008 includes a cache instancecomponent 2022 to cache one or more images to be distributed to the oneor more readers. In one embodiment, the cached images are assigned toone of the one or more readers at the web server 2008. In anotherembodiment, the images are assigned to one of the one or more readers bythe application server 2004 prior to being cached at the web server2008. The cached images are stored in a user cache memory space 2026,illustrated within the memory storage 2006 in FIG. 20.

A score client application 2024 operates on the work station 2010. Areader evaluates a scoring task item, or image, and uses the scoreclient application 2024 to create a score file that contains a score forone or more of the images. The score client 2024 uploads the score fileto the web server 2008. In one embodiment, the uploaded score file isstored in a save data folder 2028, illustrated within the memory storage2006 in FIG. 20.

The application server 2004 includes a process space 2030. A dealer2032, also referred to as an instance controller, operates in theprocess space 2030. The dealer 2032 controls the distribution of thescoring task items to the readers. A process rules application 2034 alsooperates in the process space. The process rules application 2034retrieves and applies process rules to score files, which are the filesthat are created when the reader evaluates the scoring task items at theworkstation 2010. The process rules application 2034 is capable ofspawning one or more spawn score application threads 2036 in the processspace 2030. One definition of a thread is one individual process withina single application. A thread can be defined as a sequence of computinginstructions. Each thread is capable of executing independently and on aseparate processor from the other threads. In operation, each spawnscore application thread 2036 reads the score file and sends the scorefile to the scoring rules application. The application server 2004 alsoincludes a transaction server 2038, illustrated as a MicrosoftTransaction Server (MTS), and a scoring rules application 2040 operatingon the transaction server 2038 that processes the score file and updatesa database operating on the database server 2002.

As has been provided above, the present subject matter provides systemsand methods for prioritizing the processing of standardized tests. Thepresent subject matter allows a test process to follow external timeconstraints, such as those time constraints provided by the customer.The present subject matter also allows a test process to follow internaltime constraints, such as those time constraints designed forimplementing efficient processing of the standardized tests.

This disclosure refers to several figures that resemble flow diagrams.One of ordinary skill in the art will understand, upon reading andcomprehending this disclosure, that the methods related to the flowdiagrams may occur in the order as illustrated in the flow diagrams, andmay be ordered in another manner. Thus, the present subject matter isnot limited to a particular order or logical arrangement.

Although specific embodiments have been illustrated and describedherein, it will be appreciated by those of ordinary skill in the artthat any arrangement which is calculated to achieve the same purpose maybe substituted for the specific embodiments shown. This application isintended to cover adaptations or variations of the present subjectmatter. It is to be understood that the above description is intended tobe illustrative, and not restrictive. Combinations of the aboveembodiments, and other embodiments, will be apparent to those of skillin the art upon reviewing the above description. The scope of thepresent subject matter should be determined with reference to theappended claims, along with the full scope of equivalents to which suchclaims are entitled.

1-27. (canceled)
 28. A system, comprising: means for associatingtest-processing priority information with machine-readable identifiersfor boxes of standardized test documents and with machine readableidentifiers for standardized test documents; means for readingmachine-readable identifiers on the boxes of standardized test documentsto identify the boxes; means for automatically prioritizing theidentified boxes using the associated test-processing priorityinformation; means for sorting the standardized test documents containedwithin the boxes; means for reading machine-readable identifiers on thestandardized test documents to identify the standardized test documents;means for automatically prioritizing the identified standardized testdocuments using the associated test-processing priority information; andmeans for processing the identified standardized test documents usingthe test processing priority information.
 29. The system of claim 28,further comprising: means for associating the test-processinginformation with machine-readable identifiers for standardized testitems contained in the test documents; and means for reading themachine-readable identifiers to identify the standardized test items.30. The system of claim 28, wherein the machine-readable identifiersinclude bar codes.
 31. The system of claim 28, wherein themachine-readable identifiers include electronic identifiers.
 32. Thesystem of claim 28, wherein the means for receiving test-processingpriority information includes means for receiving customer-providedtest-processing priority information.
 33. The system of claim 28,wherein the means for receiving customer-provided test-processingpriority information includes means for receiving test-processingpriority information on-line.
 34. The system of claim 28, wherein themeans for receiving test-processing priority information includes meansfor receiving test-processing priority information from atest-processing center.
 35. A system, comprising: means for receivingcriteria for prioritizing processing of test booklets; means for readingbar codes for boxes of test booklets to prioritize processing of thereceived boxes according to the received criteria; means for reading barcodes for the test booklets; means for automatically prioritizing thetest booklets according to the received criteria; and means forprocessing higher-priority test booklets before lower priority testbooklets.
 36. The system of claim 35, wherein the means for processingincludes: means for cutting the test booklets to provide pages forscanning; means for scanning the pages of the cut test booklets toprovide scanned images of the pages; means for editing the scannedimages of the pages; means for reconciling the scanned images of thepages to verify that, for each of the test booklets, the scanned imagesrepresent all pages of the test booklet; and means for validating datarepresented by the bar codes.
 37. The system of claim 35, wherein themeans for processing includes: means for assigning a plurality ofelectronic test items to a plurality of readers, each reader having oneor more credentials; and means for receiving test scores for theplurality of electronic test items from the plurality of readers. 38.The system of claim 37, wherein the means for assigning a plurality ofelectronic test items to a plurality of readers includes means forqualifying potential readers for each electronic test item from theplurality of readers, means for determining a relative priority for anelectronic test item, and means for automatically assigning scoringtasks to qualified readers such that higher-priority electronic testitems are evaluated before lower-priority electronic test items.
 39. Thesystem of claim 35, wherein the means for processing includes scoringthe test items using optical mark recognition (OMR) logic.
 40. Thesystem of claim 35, further comprising means for reporting scores forthe test items according to the received criteria.
 41. A system,comprising: means for electronically receiving a plurality of completedtest items from a plurality of test-taking centers; means for receivingcriteria for prioritizing processing of the test items; means forautomatically prioritizing the received test items according to thereceived criteria; means for processing the plurality of test items inpreparation for scoring the test items according to the receivedcriteria; and means for scoring the test items according to thecriteria.
 42. The system of claim 41, wherein the means forelectronically receiving a plurality of completed test items from aplurality of test-taking centers includes means for receiving theplurality of completed test items on-line.
 43. The system of claim 41,wherein the means for scoring the test items according to the criteriaincludes: means for assigning the plurality of electronic test items toa plurality of readers, each reader having one or more credentials; andmeans for receiving test scores for the plurality of electronic testitems from the plurality of readers.
 44. The system of claim 43, whereinthe means for assigning the plurality of electronic test items to aplurality of readers includes means for qualifying potential readers foreach electronic test item from the plurality of readers, means fordetermining a relative priority for an electronic test item, and meansfor automatically assigning scoring tasks to qualified readers such thathigher-priority electronic test items are evaluated beforelower-priority electronic test items.
 45. The system of claim 41,wherein the means for scoring the test items according to the criteriaincludes means for scoring the test items using optical mark recognition(OMR) logic.
 46. The system of claim 41, further comprising means forreporting scores for the test items according to the criteria.
 47. Asystem, comprising: at least one receiving station adapted for receivingboxes, removing test booklets from the boxes, and separating the testbooklets into sheets, wherein the boxes have machine-readable boxidentifiers, the test booklets have machine-readable bookletidentifiers, and sheets in the test booklets have machine-readable sheetidentifiers, the at least one receiving station including at least onereading device adapted to read the machine-readable box identifier, themachine-readable booklet identifier, and the machine-readable sheetidentifier; at least one scoring station adapted to process the sheetsfor a test scoring process; and a controller adapted to communicate withthe at least one receiving station and the at least one scoring station,the controller being adapted to associate priority information with themachine-readable box identifier, the machine-readable bookletidentifier, and the machine readable sheet identifier, and to determinea processing priority for each box, each booklet, and each sheetidentified using the at least one reading device.
 48. The system ofclaim 47, further comprising a report generator, the controller furtherbeing adapted to communicate with the report generator and associatereports with priority information to determine a processing priority forthe reports.
 49. The system of claim 47, further comprising acommunication module to receive priority information external timeconstraints, the priority information including the external timeconstraints.
 50. The system of claim 47, further comprising acommunication module to receive priority information external timeconstraints, the priority information including the internal timeconstraints.
 51. The system of claim 47, wherein the scoring stationincludes an optical mark recognition (OMR) scanner.
 52. The system ofclaim 47, wherein the scoring station includes a scanning stationadapted to scan scoring tasks, and a dealer adapted to distribute thescoring tasks to a human reader.